enhance: batch broadcast tombstone GC - #33
Merged
sunby merged 1 commit intoSep 11, 2026
Merged
Conversation
Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: sunby <sunbingyi1992@gmail.com>
sunby
merged commit Sep 11, 2026
cdd068f
into
codex/load-1m-segments-pr-stack-rebased-qv-work
10 of 13 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Broadcast tombstone GC still performs one catalog deletion per task after #31, so a burst of completed broadcasts can leave a long deletion backlog. This change deletes eligible tombstones in batches bounded by
metastore.maxEtcdTxnNum(default 64), preserving the existing count and lifetime retention rules.Catalog I/O holds neither task nor manager locks. A batch advances the queue and marks its tasks DONE only after deletion succeeds; failures retain the batch for idempotent retry, including an ambiguous commit result. Late ACKs ignore TOMBSTONE/DONE tasks to prevent recreating deleted records, and manager shutdown cancels in-flight GC. Existing callback ordering and durable completion remain unchanged.
Validation on Go 1.26.6:
-tags dynamic,test -gcflags="all=-N -l".-raceand the same build tags/compiler flags.run_clang_format.shcompleted.make lint-fixwas run but remains blocked by the pre-existingundefined: mocksatinternal/metastore/kv/querycoord/kv_catalog_test.go:372in the base branch.The regression tests cover retention boundaries, partial failure and retry, shutdown/recovery around deletion, late ACKs, and Load -> Release -> Load ordering through the real ACK scheduler with mocked business callbacks. A negative control removing the terminal-ACK guard fails on an unexpected catalog write.
A one-off local catalog benchmark on Go 1.26.6 / darwin-arm64 / Apple M5 deleted 4,096 pre-seeded records through the real etcd client:
These are single-sample local measurements, with seeding outside the timer; the benchmark and embedded-etcd validation used an external test overlay.
Cold-query latency and QPS still need a deployment-level rerun. TiKV ambiguous-result handling was traced through the existing wrapper but was not exercised against a live TiKV server. Sustained deletion deficits can still grow the pending queue.
Follow-up to #31.
Related: milvus-io/milvus#50549 (tombstone shutdown/recovery context).